Image processing device, imaging device, image processing method, and program

ABSTRACT

Provided is a device and a method for executing image quality improvement processing of an infrared light image. Included are: a feature amount calculating unit for receiving an infrared light image and a visible light image and extracting a feature amount from at least one of the images; and an image correcting unit for executing pixel value correction processing on the infrared light image on the basis of a reference area and a correction parameter determined depending on the feature amount. Further included are: a tap selection unit for determining the reference area used for the pixel value correction on the basis of the feature amount; and a correction parameter calculating unit for determining the correction parameter used for the pixel value correction on the basis of the feature amount. The image correcting unit executes the pixel value correction processing in which a tap determined by the tap selection unit and the correction parameter determined by the correction parameter calculating unit are applied.

TECHNICAL FIELD

The present disclosure relates to an image processing device, an imagingdevice, an image processing method, and a program. More particularly,the present disclosure relates to an image processing device, an imagingdevice, an image processing method, and a program for performing imageprocessing of a visible light image and an infrared light image.

BACKGROUND ART

Recently, systems in which an infrared light camera capable of capturinga person or other objects even in the nighttime or other situationswhere it is difficult to capture an image by visible light is combinedwith a visible light camera that captures an image of visible light arewidely used.

An imaging device using such two cameras is disclosed in, for example,Patent Document 1 (JP 2011-211387 A).

However, in general, there is a problem that infrared light cameras havelower resolution as compared to that of a visible light camera whichperforms imaging with visible light.

CITATION LIST Patent Document Patent Document 1: Japanese PatentApplication Laid-Open No. 2011-211387 SUMMARY OF THE INVENTION Problemsto be Solved by the Invention

The present disclosure has been made in view of the above problems forexample, and an object of one embodiment of the present disclosure is toprovide an image processing device, an imaging device, and an imageprocessing method, and a program for implementing higher image qualityof a captured image of an infrared light camera.

Furthermore, in an embodiment of the present disclosure, an object ofthe present invention is to provide an image processing device, animaging device, an image processing method, and a program for performingan image quality improvement processing for both a visible light imageand an infrared light image.

Solutions to Problems

A first aspect of the present disclosure is an image processing device,including:

-   -   a feature amount calculating unit for receiving an infrared        light image and a visible light image and extracting a feature        amount from at least one of the images; and    -   an image correcting unit for executing pixel value correction        processing on the infrared light image on the basis of a        reference area and a correction parameter determined depending        on the feature amount.

Furthermore, a second aspect of the present disclosure is an imagingdevice, including:

-   -   an infrared light image imaging unit for performing imaging        processing of an infrared light image;    -   a visible light image imaging unit for performing imaging        processing of a visible light image; and    -   an image processing unit for receiving the infrared light image        and the visible light image and executing pixel value correction        processing of at least one of the images,    -   in which the image processing unit includes: a feature amount        calculating unit for receiving the infrared light image and the        visible light image and extracting a feature amount from at        least one of the images; and    -   an image correcting unit for executing pixel value correction        processing on the infrared light image on the basis of a        reference area and a correction parameter determined depending        on the feature amount.

Furthermore, a third aspect of the present disclosure is an imageprocessing device, including:

-   -   a feature amount calculating unit for receiving an infrared        light image and extracting a feature amount; and    -   an image correcting unit for executing pixel value correction        processing on the infrared light image on the basis of a        reference area and a correction parameter determined depending        on the feature amount.

Furthermore, a fourth aspect of the present disclosure is an imageprocessing method executed in an image processing device, including:

-   -   a feature amount calculating step of receiving, by a feature        amount calculating unit, an infrared light image and a visible        light image and extracting a feature amount from at least one of        the images; and    -   an image correcting step of executing, by an image correcting        unit, pixel value correction processing on the infrared light        image on the basis of a reference area and a correction        parameter determined depending on the feature amount.

Furthermore, a fifth aspect of the present disclosure is a program forcausing an image processing device to execute image processing. Theprogram causes a feature amount calculating unit to receive an infraredlight image and a visible light image and to extract a feature amountfrom at least one of the images, and the program causes an imagecorrecting unit to execute pixel value correction processing on theinfrared light image on the basis of a reference area and a correctionparameter determined depending on the feature amount.

Note that the program of the present disclosure is provided in acomputer readable format to an information processing device or acomputer system that can execute various program codes, for example, andallows for provision by a storage medium or a communication medium. Byproviding such a program in a computer readable format, processingaccording to the program is implemented on the information processingdevice or the computer system.

Other objects, features, or advantages of the present disclosure willbecome clear from further detailed descriptions based on embodiments oraccompanying drawings of the present disclosure which will be describedlater. Note that in this specification, the term “system” refers to alogical group configuration of a plurality of devices, and is notlimited to those in which devices of respective components are in thesame housing.

Effects of the Invention

According to a configuration of an embodiment of the present disclosure,a device and a method for executing image quality improvement processingof an infrared light image are implemented.

Specifically, included are: a feature amount calculating unit forreceiving an infrared light image and a visible light image andextracting a feature amount from at least one of the images; and animage correcting unit for executing pixel value correction processing onthe infrared light image on the basis of a reference area and acorrection parameter determined depending on the feature amount. Furtherincluded are: a tap selection unit for determining the reference areaused for the pixel value correction on the basis of the feature amount;and a correction parameter calculating unit for determining thecorrection parameter used for the pixel value correction on the basis ofthe feature amount. The image correcting unit executes the pixel valuecorrection processing in which a tap determined by the tap selectionunit and the correction parameter determined by the correction parametercalculating unit are applied.

By these flows of processing, a device and a method for executing imagequality improvement processing of an infrared light image areimplemented.

Note that effects described herein are merely examples and thus are notlimiting. Additional effects may also be further included.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram explaining a correspondence relationship betweentypes of captured images and wavelengths of light.

FIG. 2 is a diagram explaining an example of pixel arrays of a visiblelight image and an infrared light image.

FIG. 3 is a diagram explaining processing executed by an imageprocessing device according to the present disclosure.

FIG. 4 is a diagram explaining a configuration example of the imageprocessing device.

FIG. 5 is a diagram explaining the configuration and processing of animage processing unit.

FIG. 6 is a diagram explaining types and specific examples of featureamounts.

FIG. 7 is a diagram explaining the configuration and processing of theimage processing unit that executes image correction processing in thecase where luminance distribution information is used as a featureamount.

FIG. 8 is a diagram explaining an example of image correction processingin the case where luminance distribution information is used as afeature amount.

FIG. 9 is a diagram explaining the configuration and processing of theimage processing unit that executes image correction processing in thecase where PSF information indicating a blur mode is used as a featureamount.

FIG. 10 is a diagram explaining an example of image correctionprocessing in the case where PSF information indicating a blur mode isused as a feature amount.

FIG. 11 is a diagram explaining an example of image correctionprocessing in the case where PSF information indicating a blur mode isused as a feature amount.

FIG. 12 is a diagram explaining the configuration and processing of theimage processing unit that executes image correction processing in thecase where noise information is used as a feature amount.

FIG. 13 is a diagram explaining an example of image correctionprocessing in the case where noise information is used as a featureamount.

FIG. 14 is a diagram explaining an example of image correctionprocessing in the case where noise information is used as a featureamount.

FIG. 15 is a diagram explaining the configuration and processing of animage processing unit.

FIG. 16 is a diagram explaining the configuration and processing of animage correcting unit.

FIG. 17 is a diagram explaining the configuration and processing of theimage processing device.

FIG. 18 is a diagram explaining the configuration and processing of theimage processing device.

FIG. 19 is a diagram explaining the configuration and processing of theimage processing device.

FIG. 20 is a diagram explaining the configuration and processing of theimage processing device.

FIG. 21 is a diagram illustrating a flowchart explaining an imageprocessing sequence executed by the image processing device.

FIG. 22 is a diagram explaining a hardware configuration example of theimage processing device.

MODE FOR CARRYING OUT THE INVENTION

Details of an image processing device, an imaging device, an imageprocessing method, and a program of the present disclosure will bedescribed below with reference to the drawings. Note that explanationswill be given along the following items.

-   -   1. Configuration and processing of image processing device of        the present disclosure    -   2. Image processing device for executing image quality        improvement processing of infrared light image    -   2-1. Exemplary processing of generating high quality image by        image correction processing using luminance distribution        information as image feature amount    -   2-2. Exemplary processing of generating high quality image by        image correction processing using point spread function (PSF)        (=function indicating blur mode) as image feature amount    -   2-3. Exemplary processing of generating high quality image by        image correction processing using noise information as image        feature amount    -   3. Exemplary configuration of performing image quality        improvement processing of visible light image    -   4. Other embodiments of image processing device    -   4-1. Embodiment in which reduced image of captured image is        generated and image processing is executed on reduced image    -   4-2. Embodiment in which pseudo-infrared light image based on        visible light image is generated and parallax amount and motion        information is calculated using captured infrared light image        and pseudo-infrared light image    -   4-3. Embodiment in which corrected infrared light image        generated by image correcting unit is fed back and reused    -   4-4. Embodiment in which only infrared light image is used        without using visible light image    -   5. Sequence of processing executed by image processing device    -   6. Example of hardware configuration of image processing device    -   7. Summary of configurations of the present disclosure

[1. Configuration and Processing of Image Processing Device of thePresent Disclosure]

A configuration and processing of an image processing device of thepresent disclosure will be described with reference to FIG. 1 and thesubsequent drawings.

First, an image to be processed by the image processing device of thepresent disclosure will be described with reference to FIG. 1 and thesubsequent drawings.

The image processing device of the present disclosure receives a visiblelight image and an infrared light image and executes image processing towhich these images are applied.

As illustrated in FIG. 1, a visible light image 10 is an image having awavelength in the range of about 0.4 μm to 0.7 μm, and is a color imagesuch as an RGB image captured by a general camera.

An infrared light image is an image of long wavelength light having awavelength of 0.7 μm or more. An infrared light image imaging camera forcapturing an infrared light image is capable of capturing a person orother objects generating heat in darkness or the like, for example, andis used for a surveillance camera or other cameras.

Note that, as illustrated in FIG. 1, infrared rays are classified intonear-infrared rays having a wavelength of about 0.7 to 1 μm,mid-infrared rays having a wavelength of about 3 to 5 μm, andfar-infrared rays having a wavelength of about 8 to 14 μm.

In an embodiment to be described below, an example of image processingusing an infrared light image 20 capturing mainly far-infrared rayshaving a wavelength of about 8 to 14 μm will be described.

However, the processing according to the present disclosure isapplicable not only to far-infrared light images but also to processingusing other infrared light images.

FIG. 2 is a diagram illustrating an example of pixel arrays on imagingelements that capture the visible light image 10 and the infrared lightimage 20.

A visible light image of FIG. 2(1) illustrates an example of a Bayerarray including RGB pixels. This Bayer array is used for imagingelements of many visible light imaging cameras.

Each pixel of the imaging element outputs an electric signalcorresponding to the amount of light having one of the wavelengths of R,G, and B.

On the other hand, a far-infrared light image of FIG. 2(2) captureslight having a wavelength of far-infrared light (FIR) at all pixelpositions.

However, as illustrated in (1) and (2) of FIG. 2, in general, infraredlight image imaging elements have a lower resolution than that ofvisible light image imaging elements. This is because, for example,infrared light, especially far-infrared light, has long wavelength, andthus it is difficult to use an imaging element having a high-densitypixel array.

One example of processing executed by the image processing device of thepresent disclosure will be described with reference to FIG. 3. The imageprocessing device of the present disclosure executes image processing onan infrared light image having a low resolution and generates afar-infrared light image having a high resolution, for example.

As illustrated in FIG. 3, an image processing unit 30 of the imageprocessing device according to the present disclosure inputs ahigh-resolution visible light image and a low-resolution far-infraredlight image as (A) captured images of an imaging unit.

Using these two input images, the image processing unit 30 executesimage quality improvement processing on the far-infrared light imagehaving a low resolution, and generates and outputs an image-qualityenhanced image as illustrated in (B), that is, at least one of ahigh-resolution visible light image and a high-resolution far-infraredlight image.

Hereinafter, a specific configuration and processing of the imageprocessing device that executes such image processing will be described.

[2. Image Processing Device for Executing Image Quality ImprovementProcessing of Infrared Light Image]

FIG. 4 is a block diagram illustrating a configuration of an imagingdevice which is an example of the image processing device 100 of thepresent disclosure.

Note that the image processing device according to the presentdisclosure is not limited to imaging devices but also includesinformation processing devices such as a PC that executes imageprocessing by inputting a captured image of an imaging device, forexample.

In the following, a configuration and processing of the imaging devicewill be described as an example of the image processing device 100 ofthe present disclosure.

Image processing other than imaging processing described in thefollowing embodiments is not limited to imaging devices and can beexecuted in an information processing device such as a PC.

The image processing device 100 as the imaging device illustrated inFIG. 4 includes a control unit 101, a storage unit 102, a codec 103, aninput unit 104, an output unit 105, an imaging unit 106, and an imageprocessing unit 120.

The imaging unit 106 includes an infrared light image imaging unit 107that captures a far-infrared light image and a visible light imageimaging unit 108 that captures a normal visible light image.

The infrared light image imaging unit 107 has a first imaging element111 for capturing a far-infrared light image. The first imaging element111 is an imaging element including pixels on which far-infrared lightis incident as described above with reference to FIG. 2(2) for example,and outputs an electric signal corresponding to the amount offar-infrared light incident thereon.

On the other hand, the visible light image imaging unit 108 has a secondimaging element 112 for capturing a visible light image. The secondimaging element 112 has the RGB pixels including the Bayer arraydescribed above with reference to FIG. 2(1) for example, and outputs asignal corresponding to input light of each of the colors of R, G, and Bfor each of the pixels.

The infrared light image imaging unit 107 and the visible light imageimaging unit 108 are two imaging units set at positions a predetermineddistance apart from each other, and thus captured images thereof arecaptured from different viewpoints.

The same subject image is not captured in corresponding pixels, that is,pixels at the same position, of the two images from differentviewpoints, and a subject shift corresponding to the parallax occurs.

In a case where the captured images are still images, the infrared lightimage imaging unit 107 and the visible light image imaging unit 108 eachcapture one still image, which totals two. In a case where video istaken, as a capturing frame of each of the imaging units, each of theimaging units captures continuous image frames.

Note that control of the shooting timing is performed by the controlunit 101.

The control unit 101 controls various types of processing to be executedin the imaging device 100, such as capturing an image, signal processingon captured images, image recording processing, and display processing.The control unit 101 includes a CPU or other components that executesprocessing according to various processing programs stored in thestorage unit 102, for example, and functions as a data processing unitthat executes the programs.

The storage unit 102 includes a storage unit of captured images and astorage unit for a processing program executed by the control unit 101and various parameters, as well as a RAM, a ROM, and the like thatfunction as a work area at the time of data processing.

The codec 103 executes encoding and decoding processing such ascompression and decompression processing of a captured image.

The input unit 104 is, for example, a user operation unit, and receivescontrol information such as start or end of imaging, and various modesettings.

The output unit 105 includes a display unit, a speaker, and the like andis used for displaying a captured image, a through image, etc., audiooutput, and the like.

The image processing unit 120 receives two images input from the imagingunit 106 and executes image quality improvement processing of the inputimages by applying these two images.

Specifically, for example, a corrected infrared light image 191 withimproved image quality is generated. Note that an object of the imagequality improvement processing is only the infrared light image in thisembodiment, and no image quality improvement processing is executed onthe visible light image. A visible light image 192 corresponds to thecaptured visible light image.

Note that it is also possible to perform image quality improvementprocessing on a visible light image, and such embodiment will bedescribed later.

A configuration and processing of the image processing unit 120 will bedescribed with reference to FIG. 5 and the subsequent drawings.

In the present embodiment, the image processing unit 120 receives twotypes of images of an infrared light image 201 captured by the infraredlight image imaging unit 107 and a visible light image 202 captured bythe visible light image imaging unit 108, and generates and outputs acorrected infrared light image 205 applied with image qualityimprovement processing using these two types of images.

Processing executed by the image processing unit 120 will be described.

The image processing unit 120 inputs the infrared light image 201captured by the infrared light image imaging unit 107 to a scaler 121and executes scaling processing of matching the size of the infraredlight image 201 to the size of the visible light image 202.

This is adjustment processing of image sizes for eliminating thedifference between the size of the first imaging element 111 of theinfrared light image imaging unit 107 and the size of the second imagingelement of the visible light image imaging unit 108.

In most cases, the size of the first imaging element 111 of the infraredlight image imaging unit 107 is smaller than the size of the secondimaging element of the visible light image imaging unit 108.

The scaler 121 executes scaling processing of matching the size of theinfrared light image 201 to the size of the visible light image 202.

The infrared light image 201 and the visible light image 202 sizes ofwhich are matched are input to a parallax amount detection & motiondetection unit 122 and an image position matching unit 123.

The parallax amount detection & motion detection unit 122 detects theparallax amount of the infrared light image 201 and the visible lightimage 202 and the motion amount between the two images.

The infrared light image imaging unit 107 and the visible light imageimaging unit 108 are two imaging units set at positions a predetermineddistance apart from each other, and thus captured images thereof (theinfrared light image 201 and the visible light image 202) are capturedfrom different viewpoints.

The same subject image is not captured in corresponding pixels, that is,pixels at the same position, of the two images from differentviewpoints, namely, the infrared light image 201 and the visible lightimage 202, and a subject shift corresponding to the parallax occurs.

Furthermore, in a case where these two images are not shot at perfectlythe same timing and the subject includes a moving subject, positions ofthe same subject captured in the respective images are different. Thatis, there is a motion amount of the subject.

The parallax amount detection & motion detection unit 122 detects theparallax amount between the infrared light image 201 and the visiblelight image 202 and the motion amount between the two images, and inputsthese pieces of information, namely, parallax information and motioninformation, for example a motion vector (MV), to the image positionmatching unit 123.

The image position matching unit 123 executes position matchingprocessing of the infrared light image 201 having been subjected to thesize adjustment and the visible light image 202 using the parallaxinformation and the motion information input from the parallax amountdetection & motion detection unit 122.

That is, the position matching processing of the two images is executedsuch that the same subject is located at the same position in eachimage.

Note that, specifically, processing of matching a subject position ofthe infrared light image 201 with a subject position of the visiblelight image 202 is performed for example by using the visible lightimage 202 as a reference position without moving the subject position ofthe visible light image 202.

However, which image to use as a reference image is not limited, andeither one of the images can be used as a reference image.

The image position matching unit 123 outputs two images after theposition matching, that is, a post-position matching infrared lightimage 203 and a post-position matching visible light image 204illustrated in FIG. 5 to a feature amount calculating unit 124.

The post-position matching infrared light image 203 is further input toan image correcting unit 127 via a tap selection unit 125.

The feature amount calculating unit 124 receives the post-positionmatching infrared light image 203 and the post-position matching visiblelight image 204, extracts the feature amount of images from theseimages, and outputs the extracted feature amount data to the tapselection unit 125 and a correction parameter calculating unit 126.

An example of feature amount data acquired by the feature amountcalculating unit 124 from the post-position matching infrared lightimage 203 and the post-position matching visible light image 204 will bedescribed with reference to FIG. 6.

FIG. 6 illustrates an example of the following three types of imagefeature amounts that the feature amount calculating unit 124 can extractfrom at least one of the two images.

-   -   (1) Luminance distribution information    -   (2) Point spread function (PSF) (=function indicating a blur        mode)    -   (3) Noise information

“(1) Luminance distribution information” is distribution information ofluminance values of respective pixels in an image. In the specificexample of FIG. 6(1)(b), a graph (luminance distribution graph) in whichthe pixel position is set on the horizontal axis and luminance valuesare set on the vertical axis is illustrated.

In the example illustrated in the figure, the left side of the graphcontains low luminance values while the right side contains highluminance values. Such a luminance distribution corresponds to an edgeregion such as a boundary of a subject, for example.

Note that such luminance distribution information is an image featureamount that can be acquired from only one of the post-position matchinginfrared light image 203 and the post-position matching visible lightimage 204.

“(2) Point spread function (PSF) (=function indicating a blur mode)” isa point spread function (PSF) which is a function indicating the bluramount of an image.

As illustrated in a specific example of FIG. 6(2)(b), a PSF is afunction illustrating the degree of a spread of a pixel value at acertain pixel position to the surroundings thereof, that is, the bluramount.

Note that such a point spread function is also an image feature amountthat can be acquired from only one of the post-position matchinginfrared light image 203 and the post-position matching visible lightimage 204.

“(3) Noise information” is information indicating noise included in animage. An image captured by a camera contains some noise.

In the specific example of FIG. 6(3)(b), a graph (noise distributiongraph) in which the pixel position is set on the horizontal axis andpixel values are set on the vertical axis is illustrated.

As illustrated in this graph, a pixel value is obtained by adding apredetermined amount of noise to the original color or luminance of thesubject. Note that noise includes various types of noise such as highfrequency noise and low frequency noise.

Note that such noise information is also an image feature amount thatcan be acquired from only one of the post-position matching infraredlight image 203 and the post-position matching visible light image 204.

These three image feature amounts illustrated in FIG. 6 are examples offeature amount data acquired by the feature amount calculating unit 124from at least one of the post-position matching infrared light image 203and the post-position matching visible light image 204.

The feature amount calculating unit 124 acquires at least one of thethree image feature amounts illustrated in FIG. 6 from at least one ofthe post-position matching infrared light image 203 and thepost-position matching visible light image 204.

On the basis of the acquired feature amount, the image correcting unit127 executes image correction processing as image quality improvementprocessing on the post-position matching infrared light image 203 andgenerates and outputs a corrected infrared light image 205 with improvedimage quality.

The tap selection unit 125, the correction parameter calculating unit126, and the image correcting unit 127 illustrated in FIG. 5 execute tapsetting processing and correction parameter calculating processing to beapplied to image correction processing for improving the image qualityof the post-position matching infrared light image 203 on the basis ofthe image feature amount calculated by the feature amount calculatingunit 124 as well as the image correction processing.

Hereinafter, specific processing examples of image correction processingin which the three feature amounts illustrated in FIG. 6, that is:

-   -   (1) Luminance distribution information;    -   (2) Point spread function (PSF) (=function indicating a blur        mode); and    -   (3) Noise information    -   are individually applied, that is, image quality improvement        processing of infrared light image will be described in order.

[2-1. Exemplary Processing of Generating High Quality Image by ImageCorrection Processing Using Luminance Distribution Information as ImageFeature Amount]

First, a processing example in which luminance distribution informationis acquired as an image feature amount and image quality improvementprocessing of an infrared light image is performed by image correctionprocessing using the acquired luminance distribution information will bedescribed.

FIG. 7 is a diagram illustrating a configuration similar to that of theimage processing unit 120 described with reference to FIG. 5.

A feature amount extracting unit 124 extracts luminance distributioninformation as an image feature amount, a tap selection unit 125, acorrection parameter calculating unit 126, and an image correcting unit127 execute tap setting processing and correction parameter calculatingprocessing to be applied to image correction processing for improvingthe image quality of a post-position matching infrared light image 203on the basis of the luminance distribution information extracted by thefeature amount extracting unit 124 as well as the image correctionprocessing.

FIG. 7 illustrates respective processing executed by the feature amountextracting unit 124, the tap selection unit 125, the correctionparameter calculating unit 126, and the image correcting unit 127.

As illustrated in FIG. 7, the feature amount extracting unit 124acquires the luminance distribution information as an image featureamount from the post-position matching infrared light image 203 and apost-position matching visible light image 204 in step S101.

As described above with reference to FIG. 6(1), the luminancedistribution information is distribution information of luminance valuesof respective pixels in an image. For example, the luminancedistribution information includes luminance information corresponding topixels that corresponds to a luminance distribution graph as illustratedin FIG. 6(1)(b).

The luminance distribution information extracted by the feature amountextracting unit 124 from the post-position matching infrared light image203 and the post-position matching visible light image 204 is input tothe tap selection unit 125 and the correction parameter calculating unit126.

In step S102, the tap selection unit 125 executes, on the basis of theluminance distribution information extracted by the feature amountextracting unit 124 from the post-position matching infrared light image203 and the post-position matching visible light image 204, referencepixel area selecting processing to be applied to correction processing,that is, tap selection processing.

In addition, in step S103, the correction parameter calculating unit 126calculates, on the basis of the luminance distribution informationextracted by the feature amount extracting unit 124 from thepost-position matching infrared light image 203 and the post-positionmatching visible light image 204, a correction parameter to be appliedto correction processing. For example, a multiplication coefficient tobe applied to reference pixels surrounding the current correction pixelis calculated.

An example of the tap selection processing in the tap selection unit 125and the calculation processing of the correction parameter in thecorrection parameter calculating unit 126 will be described withreference to FIG. 8.

FIG. 8 illustrates the following drawings.

-   -   (a1) Luminance distribution example of image before correction    -   (a2) Example of tap settings and correction parameter        (multiplication coefficient K_(i))    -   (b) Luminance distribution example of image after correction

(a1) In the luminance distribution example of an image beforecorrection, a luminance distribution example of a visible light imageand a luminance distribution example of an infrared light image areillustrated. Since the visible light image has been captured by a secondimaging element 112 having a high density pixel configuration arrangedin a visible light image imaging unit 108 and is a high resolutionimage, the luminance distribution reflects the luminance of a subjectmore accurately.

On the other hand, the infrared light image has been captured by a firstimaging element 111 having a low density pixel configuration arranged inan infrared light image imaging unit 107 and is also a low resolutionimage that has been subjected to enlargement processing by a scaler 121,and thus the luminance distribution is gentle without accuratelyreflecting the luminance of the subject. That is, the infrared lightimage is blurred with a low resolution.

The tap selection unit 125 performs tap selection for performing imagecorrection in which the resolution level is improved by correcting suchlow-resolution infrared light image. Specifically, a range of thereference pixels to be applied to the correction processing of pixelvalues of pixels to be corrected is set.

The tap selection unit 125 determines a reference area to be used forpixel value correction in the image correcting unit 127 on the basis ofthe feature amount input from the feature amount extracting unit 124.

Specifically, for example, a wider reference pixel area (tap area) isset as a shift in the luminance distribution information extracted fromthe post-position matching infrared light image 203 and thepost-position matching visible light image 204 is larger.

Furthermore, the correction parameter calculating unit 126 calculates acorrection parameter to be applied to the correction processing of pixelvalues of the pixels to be corrected. Specifically, a multiplicationcoefficient K_(i) to multiply pixel values of reference pixels iscalculated.

The correction parameter calculating unit 126 determines a correctionparameter to be used for pixel value correction in the image correctingunit 127 on the basis of a feature amount input from the feature amountextracting unit 124.

Specifically, for example, depending on the state of the shift in theluminance distribution information extracted from the post-positionmatching infrared light image 203 and the post-position matching visiblelight image 204, an effective correction parameter for eliminating theshift is determined.

In “(a2) Example of tap settings and correction parameter(multiplication coefficient K_(i))” in FIG. 8, with a pixel to becorrected pixel arranged in the center, positions of surroundingreference pixels used for correction of a pixel value of the currentcorrection pixel and values of the multiplication coefficient K_(i) foreach of the reference pixels are illustrated.

In the example illustrated in the drawing, 3×3=9 pixels centered at thecurrent correction pixel are illustrated. Values 0, 1, −1 illustrated inthe nine pixel positions are the multiplication coefficient K_(i) thatis the correction parameter calculated by the correction parametercalculating unit 126. Note that i is a pixel position identifierindicating a pixel position.

The tap selection unit 125 selects a pixel position referred to forcalculating a corrected pixel value of the current correction pixel asthe tap position. In the example illustrated in the figure, pixelpositions set with 1 or −1 are taps.

The correction parameter calculating unit 126 calculates amultiplication coefficient K_(i) to multiply pixel values at tappositions. These are −1 and 1 illustrated in FIG. 8(a 2).

The selection tap information set by the tap selection unit 125, thatis, the reference pixel position information and the correctionparameter calculated by the correction parameter calculating unit 126,that is, the multiplication coefficient K_(i) for a pixel value of eachreference pixel position are input to the image correcting unit 127.

The image correcting unit 127 calculates a corrected pixel value of thecurrent correction pixel on the basis of these input values.Specifically, a corrected pixel value T of the current correction pixelis calculated by application of the following equation for correctedpixel value calculation (equation 1) illustrated in step S104 in FIG. 7.

The corrected pixel value T is calculated from the following (equation1).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu} {Formula}\mspace{14mu} 1} \right\rbrack & \; \\{T = {\sum\limits_{i = 0}^{n}{{Ki} \times {Ti}}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

Note that in the above equation (1), respective symbols have thefollowing meanings.

-   -   T: Corrected pixel value of current correction pixel    -   T_(i): pixel value of reference pixel    -   i: pixel identifier of reference pixel    -   K_(i): Multiplication coefficient corresponding to reference        pixel i

The current correction pixel is, for example, a pixel at the centralposition out of the 3×3=9 pixels illustrated in FIG. 8(a 2).

Reference pixels are respective pixels of the 3×3=9 pixels, and T_(i)represent a pixel value of each of these pixels. Symbol i is anidentifier of a pixel. In the case of referring to the nine pixels, n=8is set, and a corrected pixel value T is calculated using pixel valuesof the respective pixels of i=0 to 8.

Symbol K_(i) represents a multiplication coefficient for a pixel valueT_(i) set to each pixel position i.

A pixel value of the current correction pixel is calculated from theabove (equation 1).

Note that the tap settings and settings of the correction parameter(multiplication coefficient) illustrated in FIG. 8(a 2) are merelyexamples, and settings of taps and a correction parameter are changed tovarious settings depending on a feature amount.

The image correcting unit 127 sequentially calculates corrected pixelvalues of all of the pixels included in the post-position matchinginfrared light image 203 from the above (equation 1) and generates andoutputs a corrected infrared light image 205 thereby calculated.

A luminance distribution example of the corrected infrared light image205 is illustrated in FIG. 8(b).

The luminance distribution of the corrected infrared light image 205 hasa shape closer to the luminance distribution of the visible light imageas compared with the luminance distribution of the infrared light imagebefore the correction illustrated in FIG. 8(a 1). In other words, thedistribution has improved resolution.

This is a result of performing pixel value correction for reflecting thefeature amount, that is, the luminance distribution of the visible lightimage with high resolution on pixel values of the infrared light imagewith low resolution.

In this manner, correcting pixel values of the infrared light imageusing luminance distribution information which is a feature amount ofthe visible light image with high resolution enables improvement of theimage quality of the infrared light image with low resolution. In otherwords, it becomes possible to generate and output the corrected infraredlight image 205 with an improved resolution.

[2-2. Exemplary Processing of Generating High Quality Image by ImageCorrection Processing Using Point Spread Function (PSF) (=FunctionIndicating Blur Mode) as Image Feature Amount]

Next, an example of processing of performing an image qualityimprovement processing of an infrared light image by acquiring the pointspread function (PSF) (=function indicating a mode indicating a blurmode) as an image feature amount and performing image correctionprocessing using the acquired point spread function (PSF) informationwill be described.

Like FIG. 7, FIG. 9 is a diagram illustrating a configuration similar tothat of the image processing unit 120 described with reference to FIG.5.

A feature amount extracting unit 124 extracts a point spread function(PSF) (=function indicating a mode indicating a blur mode) as an imagefeature amount from a post-position matching infrared light image 203and outputs the PSF function to a tap selection unit 125 and acorrection parameter calculating unit 126.

The tap selection unit 125, the correction parameter calculating unit126, and an image correcting unit 127 execute tap setting processing andcorrection parameter calculating processing to be applied to imagecorrection processing for improving the image quality of thepost-position matching infrared light image 203 on the basis of thepoint spread function (PSF) (=function indicating a mode indicating ablur mode) extracted by the feature amount extracting unit 124 from thepost-position matching infrared light image 203 as well as the imagecorrection processing.

FIG. 9 illustrates respective processing executed by the feature amountextracting unit 124, the tap selection unit 125, the correctionparameter calculating unit 126, and the image correcting unit 127.

As illustrated in FIG. 9, in step S121, the feature amount extractingunit 124 acquires a point spread function (PSF) (=function indicating amode indicating a blur mode) as an image feature amount from thepost-position matching infrared light image 203.

The point spread function (PSF) (=function indicating the blur mode) isa function indicating the blur amount of an image as described abovewith reference to FIG. 6(2).

As illustrated in a specific example of FIG. 6(2)(b), a PSF is afunction illustrating the degree of a spread of a pixel value at acertain pixel position to the surroundings thereof, that is, the bluramount.

Note that in this case the point spread function (PSF) is acquired usingthe post-position matching infrared light image 203.

The point spread function (PSF) information extracted from thepost-position matching infrared light image 203 by the feature amountextracting unit 124 is input to the tap selection unit 125 and thecorrection parameter calculating unit 126.

In step S122, the tap selection unit 125 executes, on the basis of thepoint spread function (PSF) information extracted by the feature amountextracting unit 124 from the post-position matching infrared light image203, reference pixel area selecting processing to be applied tocorrection processing, that is, tap selection processing.

Specifically, for example, a wider reference pixel area (tap area) isset as a blur amount in the post-position matching infrared light image203 is larger.

In addition, in step S123, the correction parameter calculating unit 126calculates, on the basis of the point spread function (PSF) informationextracted by the feature amount extracting unit 124 from thepost-position matching infrared light image 203, a correction parameterto be applied to correction processing.

Specifically, a coefficient for forming an inverse filter which is afilter for eliminating blur, that is, a multiplication coefficient to beapplied to reference pixels surrounding the current correction pixel iscalculated.

An example of the tap selection processing in the tap selection unit 125and the calculation processing of the correction parameter in thecorrection parameter calculating unit 126 will be described withreference to FIGS. 10 and 11.

FIG. 10 illustrates the following drawings.

-   -   (a1) Pixel value distribution example of image before correction    -   (a2) Example of tap settings and correction parameter        (multiplication coefficient K_(i))    -   (b) Pixel value distribution example of image after correction

Item (a1) Pixel value distribution example of image before correctionincludes a pixel value distribution example of an infrared light imageto be corrected.

As described above, the infrared light image has been captured by afirst imaging element 111 having a low density pixel configurationarranged in an infrared light image imaging unit 107 and is also a lowresolution image that has been subjected to enlargement processing by ascaler 121, and thus a pixel value distribution thereof smoothlyreflects the luminance of the subject. That is, the infrared light imagehas large blur.

The tap selection unit 125 performs tap selection for performing imagecorrection in which the infrared light image having such large blur iscorrected such that a clear image with less blur is obtained.Specifically, a range of the reference pixels to be applied to thecorrection processing of pixel values of pixels to be corrected is set.

The tap selection unit 125 determines a reference area to be used forpixel value correction in the image correcting unit 127 on the basis ofthe feature amount input from the feature amount extracting unit 124.

Specifically, for example, a wider reference pixel area (tap area) isset as a blur amount in the post-position matching infrared light image203 is larger.

Furthermore, the correction parameter calculating unit 126 calculates acorrection parameter to be applied to the correction processing of pixelvalues of the pixels to be corrected. Specifically, a coefficient forforming an inverse filter which is a filter for eliminating blur, thatis, a multiplication coefficient K_(i) to be applied to reference pixelssurrounding the current correction pixel is calculated.

The correction parameter calculating unit 126 determines a correctionparameter to be used for pixel value correction in the image correctingunit 127 on the basis of a feature amount input from the feature amountextracting unit 124.

Specifically, for example, an effective correction parameter foreliminating the blur of the post-position matching infrared light image203 is determined.

In “(a2) Example of tap settings and correction parameter(multiplication coefficient K_(i))” in FIG. 10, with a pixel to becorrected pixel arranged in the center, positions of surroundingreference pixels used for correction of a pixel value of the currentcorrection pixel and values of the multiplication coefficient K_(i) foreach of the reference pixels are illustrated.

In the example illustrated in the drawing, 3×3=9 pixels centered at thecurrent correction pixel are illustrated. Values 0, −1, and 9illustrated in the nine pixel positions are the multiplicationcoefficient K_(i) that is the correction parameter calculated by thecorrection parameter calculating unit 126. Note that i is a pixelposition identifier indicating a pixel position.

The tap selection unit 125 selects a pixel position referred to forcalculating a corrected pixel value of the current correction pixel asthe tap position. In the example illustrated in the figure, pixelpositions set with −1 or 9 are taps.

The correction parameter calculating unit 126 calculates amultiplication coefficient K_(i) to multiply pixel values at tappositions. These are −1 and 9 illustrated in FIG. 10(a 2).

The selection tap information set by the tap selection unit 125, thatis, the reference pixel position information and the correctionparameter calculated by the correction parameter calculating unit 126,that is, the multiplication coefficient K_(i) for a pixel value of eachreference pixel position are input to the image correcting unit 127.

The image correcting unit 127 calculates a corrected pixel value of thecurrent correction pixel on the basis of these input values.Specifically, a corrected pixel value T of the current correction pixelis calculated by application of the following equation for correctedpixel value calculation (equation 2) illustrated in step S124 in FIG. 9.

The corrected pixel value T is calculated from the following (equation2).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu} {Formula}\mspace{14mu} 2} \right\rbrack & \; \\{T = {\sum\limits_{i = 0}^{n}{{Ki} \times {Ti}}}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

Note that in the above (equation 2), respective symbols have thefollowing meanings.

-   -   T: Corrected pixel value of current correction pixel    -   T_(i): pixel value of reference pixel    -   i: pixel identifier of reference pixel    -   K_(i): Multiplication coefficient corresponding to reference        pixel i

The current correction pixel is, for example, a pixel at the centralposition out of the 3×3=9 pixels illustrated in FIG. 10(a 2).

Reference pixels are respective pixels of the 3×3=9 pixels, and T_(i)represent a pixel value of each of these pixels. Symbol i is anidentifier of a pixel. In the case of referring to the nine pixels, n=8is set, and a corrected pixel value T is calculated using pixel valuesof the respective pixels of i=0 to 8.

Symbol K_(i) represents a multiplication coefficient for a pixel valueT_(i) set to each pixel position i.

A pixel value of the current correction pixel is calculated from theabove (equation 2).

Note that the tap settings and settings of the correction parameter(multiplication coefficient) illustrated in FIG. 10(a 2) are mereexamples, and settings of taps and a correction parameter are changed tovarious settings depending on a feature amount.

The image correcting unit 127 sequentially calculates corrected pixelvalues of all of the pixels included in the post-position matchinginfrared light image 203 from the above (equation 2) and generates andoutputs a corrected infrared light image 205 thereby calculated.

A pixel value distribution example of the corrected infrared light image205 is illustrated in FIG. 10(b).

With the pixel value distribution of the corrected infrared light image205, as compared with the pixel value distribution of the infrared lightimage before correction illustrated in FIG. 10(a 1), the image has asteeper gradient in the pixel value variation with eliminated blur.

This is a result of performing the pixel value correction in which aninverse filter set with the coefficient is applied as a blur eliminationfilter.

In this manner, correcting pixel values of the infrared light imageusing the PSF information which is a feature amount indicating a blurmode of the infrared light image enables improvement of the imagequality of the infrared light image with much blur. In other words, itbecomes possible to generate and output the corrected infrared lightimage 205 with a reduced blur amount.

Note that the example illustrated in (a1) and (a2) of FIG. 10 is anexample of tap settings and settings of a correction parameter(multiplication coefficient K_(i)) in a case where a spread of blur isrelatively narrow.

The tap settings and setting of the correction parameter (multiplicationcoefficient K_(i)) are changed depending on the PSF acquired as afeature amount, that is, the blur mode.

FIG. 11 illustrates an example of tap settings and settings of acorrection parameter (multiplication coefficient K_(i)) in a case wherea spread of blur is relatively wide.

As illustrated in (a1) and (a2) of FIG. 11, in a case where a bluramount is large, processing is performed such that corrected pixelvalues are determined on the basis of pixel values of reference pixelsin a wider range by using tap settings that allow a pixel area to bereferred to for determination to be larger.

In this manner, by executing the reference pixel area selectingprocessing to be applied to the correction processing, that is, the tapselection processing on the basis of the point spread function (PSF)information calculated by the feature amount calculating unit 124 andperforming correction processing by calculating the correction parameter(multiplication coefficient), optimum pixel value correction dependingon a blur mode becomes possible, and generation of a high qualitycorrected infrared light image with reduced blur becomes possible.

[2-3. Exemplary Processing of Generating High Quality Image by ImageCorrection Processing Using Noise Information as Image Feature Amount]

Next, a processing example in which noise information is acquired as animage feature amount and image quality improvement processing of aninfrared light image is performed by image correction processing usingthe acquired noise information will be described.

Like FIGS. 7 and 9, FIG. 12 is a diagram illustrating a configurationsimilar to that of the image processing unit 120 described withreference to FIG. 5.

A feature amount extracting unit 124 extracts noise information as animage feature amount from a post-position matching visible light image204 and outputs the post-position matching visible light image 204 to atap selection unit 125 and a correction parameter calculating unit 126.

The tap selection unit 125, the correction parameter calculating unit126, and an image correcting unit 127 execute tap setting processing andcorrection parameter calculating processing to be applied to imagecorrection processing for improving the image quality of thepost-position matching infrared light image 203 on the basis of thenoise information extracted by the feature amount extracting unit 124from the post-position matching visible light image 204 as well as theimage correction processing.

FIG. 12 illustrates respective processing executed by the feature amountextracting unit 124, the tap selection unit 125, the correctionparameter calculating unit 126, and the image correcting unit 127.

As illustrated in FIG. 12, the feature amount extracting unit 124acquires the noise information as an image feature amount from thepost-position matching visible light image 204 in step S141.

As described above with reference to FIG. 6(3), the noise informationindicates noise included in the image. An image captured by a cameracontains some noise.

As illustrated in the specific example of FIG. 6(3)(b), pixel values setin the image includes a predetermined amount of noise, and a pixel valueis obtained by adding a predetermined amount of noise to the originalcolor or luminance of the subject. Note that noise includes varioustypes of noise such as high frequency noise and low frequency noise.

Note that Here, the noise information is acquired using thepost-position matching visible light image 204.

The noise information extracted from the post-position matching visiblelight image 204 by the feature amount extracting unit 124 is input tothe tap selection unit 125 and the correction parameter calculating unit126.

In step S142, the tap selection unit 125 executes, on the basis of thenoise information extracted by the feature amount extracting unit 124from the post-position matching visible light image 204, reference pixelarea selecting processing to be applied to correction processing, thatis, tap selection processing.

Specifically, for example, a wider reference pixel area (tap area) isset as noise included the post-position matching visible light image 204includes more noise having a low frequency band component.

In addition, in step S143, the correction parameter calculating unit 126calculates, on the basis of the noise information extracted by thefeature amount extracting unit 124 from the post-position matchingvisible light image 204, a correction parameter to be applied tocorrection processing.

Specifically, a multiplication coefficient to be applied to referencepixels surrounding the current correction pixel is calculated.

An example of the tap selection processing in the tap selection unit 125and the calculation processing of the correction parameter in thecorrection parameter calculating unit 126 will be described withreference to FIGS. 13 and 14.

FIG. 13 illustrates the following drawings.

-   -   (a1) Pixel value distribution example of image before correction    -   (a2) Example of tap settings and correction parameter        (multiplication coefficient K_(i))    -   (b) Pixel value distribution example of image after correction

Item (a1) Pixel value distribution example of image before correctionincludes a pixel value distribution example of an infrared light imageto be corrected.

As described above, the infrared light image has been captured by afirst imaging element 111 having a low density pixel configurationarranged in an infrared light image imaging unit 107 and is also a lowresolution image that has been subjected to enlargement processing by ascaler 121, and thus a pixel value distribution thereof smoothlyreflects the luminance of the subject. The image includes more noisethan a visible light image which is a high resolution image.

The tap selection unit 125 performs tap selection for performing imagecorrection in which the infrared light image including much noise iscorrected such that an image with less noise is obtained. Specifically,a range of the reference pixels to be applied to the correctionprocessing of pixel values of pixels to be corrected is set.

The tap selection unit 125 determines a reference area to be used forpixel value correction in the image correcting unit 127 on the basis ofthe feature amount input from the feature amount extracting unit 124.

Specifically, for example, a wider reference pixel area (tap area) isset as noise in the post-position matching visible light image 204includes more noise having a low frequency band component.

Furthermore, the correction parameter calculating unit 126 calculates acorrection parameter to be applied to the correction processing of pixelvalues of the pixels to be corrected. Specifically, a coefficient forforming a filter for reducing noise, that is, a multiplicationcoefficient K_(i) to be applied to reference pixels surrounding thecurrent correction pixel is calculated.

The correction parameter calculating unit 126 determines a correctionparameter to be used for pixel value correction in the image correctingunit 127 on the basis of a feature amount input from the feature amountextracting unit 124.

Specifically, for example, a correction parameter effective for noisereduction is determined depending on a noise component included in thepost-position matching visible light image 204.

In “(a2) Example of tap settings and correction parameter(multiplication coefficient K_(i))” in FIG. 13, with a pixel to becorrected pixel arranged in the center, positions of surroundingreference pixels used for correction of a pixel value of the currentcorrection pixel and values of the multiplication coefficient K_(i) foreach of the reference pixels are illustrated.

In the example illustrated in the drawing, 3×3=9 pixels centered at thecurrent correction pixel are illustrated. Values 1/9 illustrated in thenine pixel positions are the multiplication coefficient K_(i) that isthe correction parameter calculated by the correction parametercalculating unit 126. Note that i is a pixel position identifierindicating a pixel position.

Note that these coefficient settings correspond to coefficient settingsof a smoothing filter.

The tap selection unit 125 selects a pixel position referred to forcalculating a corrected pixel value of the current correction pixel asthe tap position. In the example illustrated in the figure, all the ninepixel positions set with 1/9 are taps.

The correction parameter calculating unit 126 calculates amultiplication coefficient K_(i) to multiply pixel values at tappositions. That is, 1/9 illustrated in FIG. 13(a 2).

The selection tap information set by the tap selection unit 125, thatis, the reference pixel position information and the correctionparameter calculated by the correction parameter calculating unit 126,that is, the multiplication coefficient K_(i) for a pixel value of eachreference pixel position are input to the image correcting unit 127.

The image correcting unit 127 calculates a corrected pixel value of thecurrent correction pixel on the basis of these input values.Specifically, a corrected pixel value T of the current correction pixelis calculated by application of the following equation for correctedpixel value calculation (equation 3) illustrated in step S144 in FIG.12.

The corrected pixel value T is calculated from the following (equation3).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu} {Formula}\mspace{14mu} 3} \right\rbrack & \; \\{T = {\sum\limits_{i = 0}^{n}{{Ki} \times {Ti}}}} & \left( {{Equation}\mspace{14mu} 3} \right)\end{matrix}$

Note that in the above (equation 3), respective symbols have thefollowing meanings.

-   -   T: Corrected pixel value of current correction pixel    -   T_(i): pixel value of reference pixel    -   i: pixel identifier of reference pixel    -   K_(i): Multiplication coefficient corresponding to reference        pixel i

The current correction pixel is, for example, a pixel at the centralposition out of the 3×3=9 pixels illustrated in FIG. 13(a 2).

Reference pixels are respective pixels of the 3×3=9 pixels, and T_(i)represent a pixel value of each of these pixels. Symbol i is anidentifier of a pixel. In the case of referring to the nine pixels, n=8is set, and a corrected pixel value T is calculated using pixel valuesof the respective pixels of i=0 to 8.

Symbol K_(i) represents a multiplication coefficient for a pixel valueT_(i) set to each pixel position i.

A pixel value of the current correction pixel is calculated from theabove (equation 3).

Note that the tap settings and settings of the correction parameter(multiplication coefficient) illustrated in FIG. 13(a 2) are mereexamples, and settings of taps and a correction parameter are changed tovarious settings depending on a feature amount.

The image correcting unit 127 sequentially calculates corrected pixelvalues of all of the pixels included in the post-position matchinginfrared light image 203 from the above (equation 3) and generates andoutputs a corrected infrared light image 205 thereby calculated.

A pixel value distribution example of the corrected infrared light image205 is illustrated in FIG. 13(b).

With the pixel value distribution of the corrected infrared light image205, as compared with the pixel value distribution of the infrared lightimage before correction illustrated in FIG. 13(a 1), the image hasreduced noise included in the pixel values.

This is a result of performing the pixel value correction in which, forexample, a smoothing filter set with a coefficient is applied as a noiseelimination filter.

In this manner, correcting pixel values of the infrared light imageusing the noise information which is a feature amount indicating a noisemode of the infrared light image enables improvement of the imagequality of the infrared light image with much noise. In other words, itbecomes possible to generate and output the corrected infrared lightimage 205 with a reduced noise amount.

Note that the example illustrated in (a1) and (a2) of FIG. 13 is anexample of tap settings and settings of a correction parameter(multiplication coefficient K_(i)) in a case where a large part of noiseincluded in the infrared light image is noise having a high frequencyband component and there is relatively small amount of noise having alow frequency band component.

The tap settings and setting of the correction parameter (multiplicationcoefficient K_(i)) are changed depending on the noise informationacquired as a feature amount.

FIG. 14 illustrates an example of tap settings and settings of acorrection parameter (multiplication coefficient K_(i)) in the casewhere there are more noise having a low frequency band component ascompared with the example of FIG. 13.

As illustrated in (a1) and (a2) of FIG. 14, in a case where noise havinga low frequency band component is large, processing is performed suchthat corrected pixel values are determined on the basis of pixel valuesof reference pixels in a wider range by using tap settings that allow apixel area to be referred to for determination to be larger.

In this manner, by executing the reference pixel area selectingprocessing to be applied to the correction processing, that is, the tapselection processing on the basis of the noise information calculated bythe feature amount calculating unit 124 and performing correctionprocessing by calculating the correction parameter (multiplicationcoefficient), optimum pixel value correction depending on a noise modebecomes possible, and generation of a high quality corrected infraredlight image with reduced noise becomes possible.

Note that in the embodiments of the respective image processing devicesdescribed above, an example of performing image correction processing inwhich the three feature amounts illustrated in FIG. 6, that is:

-   -   (1) Luminance distribution information;    -   (2) Point spread function (PSF) (=function indicating a blur        mode); and    -   (3) Noise information    -   are individually applied, that is, image quality improvement        processing of an infrared light image has been described.

As described above, it is possible to perform image correction using onetype of feature amount. However, image correction may be performed bycombining any two or three of the above feature amounts (1) to (3).

[3. Exemplary Configuration of Performing Image Quality ImprovementProcessing of Visible Light Image]

The image processing devices described with reference to FIGS. 5 to 14execute image quality improvement processing of an infrared light imageonly.

Next, with reference to FIG. 15 and the subsequent drawings, aconfiguration and processing of an image processing device that executesimage quality improvement processing also on a visible light image inaddition to an infrared light image will be described.

FIG. 15 is a diagram illustrating a configuration example of an imageprocessing device of this embodiment. The configuration illustrated inFIG. 15 has a substantially similar configuration to that of FIG. 5described above. A different point is that a post-position matchingvisible light image 204 output from an image position matching unit 123is input to an image correcting unit 127 and that the image correctingunit 127 performs image correction as image quality improvementprocessing on the post-position matching visible light image 204 andoutputs a corrected visible light image 206.

The rest of the configuration is similar to the configurations describedwith reference to FIG. 5 and the subsequent drawings, and as for aninfrared light image, similar image quality improvement processing tothose described with reference to FIGS. 5 to 14 is performed thereon,and a corrected infrared light image 205 is thereby output.

That is, in the present embodiment, the image correcting unit 127executes, on an infrared light image, similar quality improvementprocessing to those described with reference to FIGS. 5 to 14 andfurther executes, on a visible light image, image correction processingas image quality improvement processing.

The configuration and processing of the image correcting unit 127 willbe described with reference to FIG. 16.

As illustrated in FIG. 16, the image correcting unit 127 includes aninfrared light image correcting unit 131 and a visible light imagecorrecting unit 132.

The infrared light image correcting unit 131 executes image correctionprocessing based on tap settings and correction parameter settingssimilar to those described above with reference to FIGS. 5 to 14 andgenerates and outputs the corrected infrared light image 205 withimproved image quality.

In the present embodiment, the corrected infrared light image 205 isfurther input to the visible light image correcting unit 132.

The visible light image correcting unit 132 executes superimpositionprocessing of the corrected infrared light image 205 on thepost-position matching visible light image 204. Specifically, forexample, blending processing of pixel values at corresponding positions(same coordinate positions) of the post-position matching visible lightimage 204 and the corrected infrared light image 205, in other words,for example, processing of adding high frequency band components of thecorrected infrared light image 205 to the post-position matching visiblelight image 204 is executed, and thereby the corrected visible lightimage 206 is generated and output.

The corrected infrared light image 205 is the image quality of which hasbeen improved by the processing described above with reference to FIGS.5 to 14, and by blending pixel values set in the image with the improvedquality with the original pixel values of the post-position matchingvisible light image 204, a corrected image of the visible light image isgenerated.

By this processing, the corrected visible light image 206 with improvedimage quality is generated and output.

[4. Other Embodiments of Image Processing Device]

Next, other embodiments of the image processing device will be describedwith reference to FIG. 17 and the subsequent drawings.

The following embodiments will be described in order.

-   -   (1) Embodiment in which reduced image of captured image is        generated and image processing is executed on reduced image    -   (2) Embodiment in which pseudo-infrared light image based on        visible light image is generated and parallax amount and motion        information is calculated using captured infrared light image        and pseudo-infrared light image    -   (3) Embodiment in which corrected infrared light image generated        by image correcting unit is fed back and reused    -   (4) Embodiment in which only infrared light image is used        without using visible light image

[4-1. Embodiment in which Reduced Image of Captured Image is Generatedand Image Processing is Executed on Reduced Image]

First, an embodiment in which reduced image of captured image isgenerated and image processing is executed on the reduced image isdescribed with reference to FIG. 17.

FIG. 17 is a diagram illustrating a configuration of an image processingdevice of this embodiment.

The configuration illustrated in FIG. 17 has a substantially similarconfiguration to that of FIG. 5 described above. Different points arethat an image reducing unit 151 is added and that an image correctingunit 127 executes image enlargement processing.

The rest of the configuration is similar to the configurations describedwith reference to FIGS. 5 and 14, and as for an infrared light image,similar image quality improvement processing to those described withreference to FIGS. 5 to 14 is performed thereon, and a correctedinfrared light image 205 is thereby output.

In the image processing device illustrated in FIG. 17, the size of avisible light image 202 captured by a visible light image imaging unit108 is reduced in the image reducing unit 151.

Furthermore, a scaler 121 executes processing of reducing the size of aninfrared light image 201 captured by an infrared light image imagingunit 107 to the same size as the visible light size having been reducedin the image reducing unit 151.

In the subsequent processing, that is, in a parallax amount detection &motion detection unit 122, an image position matching unit 123, afeature amount calculating unit 124, a tap selection unit 125, acorrection parameter calculating unit 126, and the image correcting unit127, processing is executed using this reduced image.

Finally, the image correcting unit 127 executes enlargement processingof enlarging the infrared light image with high image quality generatedon the basis of the reduced image tor the original size or a similarimage size to the original size of the visible light image and therebyoutputs the enlarged infrared light image as the corrected infraredlight image 205.

In this embodiment, the parallax amount detection & motion detectionunit 122, the image position matching unit 123, the feature amountcalculating unit 124, the tap selection unit 125, the correctionparameter calculating unit 126, and the image correcting unit 127execute the processing using the reduced image, thereby enablingreduction in processing time and resources such as a memory and a dataprocessing unit necessary for the processing.

Thus, this allows reliable processing to be performed at a high speedeven in a mobile terminal or other devices with limited data processingfunction or a small memory capacity.

[4-2. Embodiment in which Pseudo-Infrared Light Image Based on VisibleLight Image is Generated and Parallax Amount and Motion Information areCalculated Using Captured Infrared Light Image and Pseudo-Infrared LightImage]

Next, an embodiment in which a pseudo-infrared light image based on avisible light image is generated and the parallax amount and motioninformation are calculated using the captured infrared light image andthe pseudo-infrared light image will be described with reference to FIG.18.

FIG. 18 is a diagram illustrating a configuration of an image processingdevice of this embodiment.

The configuration illustrated in FIG. 18 has a substantially similarconfiguration to that of FIG. 5 described above. A different point isthat a pseudo-infrared light image generating unit 161 is added.

The rest of the configuration is similar to the configurations describedwith reference to FIGS. 5 and 14, and as for an infrared light image,similar image quality improvement processing to those described withreference to FIGS. 5 to 14 is performed thereon, and a correctedinfrared light image 205 is thereby output.

In the image processing device illustrated in FIG. 18, a visible lightimage 202 captured by a visible light image imaging unit 108 is input tothe pseudo-infrared light image generating unit 161, and thepseudo-infrared light image generating unit 161 generates apseudo-infrared light image based on the visible light image 202.

The visible light image 202 has, for example, a pixel array according tothe Bayer array including the respective RGB pixels described above withreference to FIG. 2(1).

The pseudo-infrared light image generating unit first executes demosaicprocessing based on the RGB array image and sets a G pixel value toevery pixel. Furthermore, on the basis of the G pixel value, apseudo-infrared light pixel value (IR) is calculated.

For example, an infrared light pixel value (IR_(i)) of each pixel iincluded in an image is calculated from the following equation.

IR_(i) =a×G _(i)

Note that the above example using the G pixel value is merely anexample, and the pseudo-infrared light image generating unit 161 maygenerate the pseudo-infrared light image 161 based on the visible lightimage 202 by applying another method.

In the above equation:

-   -   symbol i represents an identifier of a pixel;    -   symbol IR_(i) represents a pseudo-infrared light pixel value of        a pixel i of the pseudo-infrared light image;    -   symbol a represents a predetermined coefficient; and    -   G_(i) represents a pixel value of a pixel i of the image        generated by the demosaic processing of the visible light image.

The pseudo-infrared light image generated by the pseudo-infrared lightimage generating unit 161 on the basis of the visible light image 202 isinput to the parallax amount detection & motion detection unit 122together with the infrared light image captured by an infrared lightimage imaging unit 107 and adjusted to the same size as the visiblelight image 202 by a scaler 121.

The parallax amount detection & motion detection unit 122 compares thesetwo input images and detects the parallax amount and the motion amountbetween the images.

In this embodiment, the parallax amount detection & motion detectionunit 122 is configured to detect the parallax amount and the motionamount between images by using two images having the same quality whichare an infrared light image and a pseudo-infrared light image, therebyenabling detection processing with higher accuracy.

Subsequent processing by an image position matching unit 123, a featureamount calculating unit 124, a tap selection unit 125, a correctionparameter calculating unit 126, and an image correcting unit 127 issimilar to that described above with reference to FIGS. 5 to 14. Theinfrared light image is subjected to image quality improvementprocessing similar to that described with reference to FIGS. 5 to 14,and thereby the corrected infrared light image 205 is output.

[4-3. Embodiment in which Corrected Infrared Light Image Generated byImage Correcting Unit is Fed Back and Reused]

Next, an embodiment in which a corrected infrared light image generatedby an image correcting unit is fed back and reused will be describedwith reference to FIG. 19.

FIG. 19 is a diagram illustrating a configuration of an image processingdevice of this embodiment.

The configuration illustrated in FIG. 19 has a substantially similarconfiguration to that of FIG. 5 described above. A different point isthat a subtractor 171 and an adder 172 are added.

In the image processing device illustrated in FIG. 19, a post-positionmatching infrared light image 203 after position matching processing inthe image position matching unit 123 is input to the subtractor 171.

In the subtractor 171, a differential image of a post-position matchingcorrected infrared light image 211 is generated from the post-positionmatching infrared light image 203.

The post-position matching corrected infrared light image 211 is animage after position matching processing obtained by adjusting eachpixel position of a corrected infrared light image 205 generated on thebasis of a previous image frame to the same pixel position as that ofthe post-position matching infrared light image 203.

The differential image output from the subtractor 171 is output to afeature amount calculating unit 124 and further output to an imagecorrecting unit 127 via a tap selection unit 125.

Processing in the feature amount calculating unit 124, the tap selectionunit 125, and the image correcting unit 127 is executed on thedifferential image.

The differential image output from the image correcting unit 127 isinput to the adder 172, the adder 1272 executes addition processing ofthe corrected differential image and the post-position matchingcorrected infrared light image, and the result is output as a correctedinfrared light image 205.

Note that the corrected infrared light image 205 is also fed back to theimage position matching unit 123 and applied to processing of a nextimage frame.

In this embodiment, the correction processing is performed in advance,and a preceding image frame which is an image having an improved qualityis fed back in correction processing of a next image to be corrected.

This configuration enables performance of correction processing of anext image frame by referring to the preceding correction result,thereby allowing the correction accuracy to be improved.

[4-4. Embodiment in which Only Infrared Light Image is Used withoutUsing Visible Light Image]

Next, an embodiment in which only an infrared light image is usedwithout using a visible light image will be described with reference toFIG. 20.

FIG. 20 is a diagram illustrating a configuration of an image processingdevice of this embodiment.

Unlike the image processing device illustrated in FIG. 5 describedabove, the image processing device illustrated in FIG. 20 does not havethe visible light image imaging unit 108.

Only an infrared light image imaging unit 107 is included, and imageprocessing using only an infrared light image 201 is executed.

Processing executed by the image processing unit 120 will be described.

The image processing unit 120 inputs the infrared light image 201captured by the infrared light image imaging unit 107 to a featureamount calculating unit 181 and to an image correcting unit 184 via atap selection unit 182.

The feature amount calculating unit 181 extracts a feature amount of theimage from the infrared light image 201, and outputs the extractedfeature amount data to the tap selection unit 182 and a correctionparameter calculating unit 183.

The feature amount data acquired from the infrared light image 201 bythe feature amount calculating unit 181 is, for example, a featureamount described above with reference to FIG. 6 and includes at leastany one of the following three types of image feature amounts.

-   -   (1) Luminance distribution information    -   (2) Point spread function (PSF) (=function indicating a blur        mode)    -   (3) Noise information

The feature amount calculation 181 acquires at least one of these threeimage feature amounts. On the basis of the acquired feature amount, theimage correcting unit 184 executes image correction processing as imagequality improvement processing on the infrared light image 201 andgenerates and outputs a corrected infrared light image 205 with improvedimage quality.

The tap selection unit 182, the correction parameter calculating unit183, and the image correcting unit 184 illustrated in FIG. 20 executetap setting processing and correction parameter calculating processingto be applied to image correction processing for improving the imagequality of the infrared light image 201 on the basis of the imagefeature amount calculated by the feature amount calculating unit 181 aswell as the image correction processing.

These flows of processing have similar configurations to the processingdescribed above with reference to FIGS. 5 to 14.

For example, the image correcting unit 184 calculates a corrected pixelvalue T of the current correction pixel by applying the followingequation for corrected pixel value calculation (equation 4) illustratedin FIG. 20.

The corrected pixel value T is calculated from the following (equation4).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu} {Formula}\mspace{14mu} 4} \right\rbrack & \; \\{T = {\sum\limits_{i = 0}^{n}{{Ki} \times {Ti}}}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$

Note that in the above (equation 4), respective symbols have thefollowing meanings.

-   -   T: Corrected pixel value of current correction pixel    -   T_(i): pixel value of reference pixel i: pixel identifier of        reference pixel K_(i): Multiplication coefficient corresponding        to reference pixel i

This embodiment uses only the infrared light image and thus isapplicable to, for example, a monocular type imaging device.

[5. Sequence of Processing Executed by Image Processing Device]

Next, a processing sequence executed by an image processing deviceaccording to the present disclosure will be described.

FIG. 21 is a flowchart explaining a processing sequence executed in theimage processing units of the image processing devices described withreference to FIGS. 5 to 14.

Processing according to the flow illustrated in FIG. 21 is executed, forexample, under the control of a control unit having a program executionfunction according to a program stored in a storage unit of the imageprocessing device.

Hereinafter, processing of each step of the flow illustrated in FIG. 21will be sequentially described.

(Steps S301 a and 301 b)

Steps S301 a and 301 b represent image capturing processing.

For example, images are captured by the infrared light image imagingunit 107 and the visible light image imaging unit 108 illustrated inFIG. 5.

Step S301 a is imaging processing of the visible light image 202 in thevisible light image imaging unit 108 illustrated in FIG. 5.

Step S301 b is imaging processing of the infrared light image 201 in theinfrared light image imaging unit 107.

(Step S302)

Step S302 is processing executed by the scaler 121 illustrated in FIG.5.

In step S302, the scaler 121 inputs the infrared light image 201captured by the infrared light image imaging unit 107 in step S301 b andexecutes scaling processing of adjusting the size of the infrared lightimage 201 to the size of the visible light image 202.

The infrared light image 201 and the visible light image 202 sizes ofwhich are matched are input to a parallax amount detection & motiondetection unit 122 and an image position matching unit 123.

(Step S303)

Step S303 is processing executed by the parallax amount detection &motion detection unit 122 illustrated in FIG. 5.

The parallax amount detection & motion detection unit 122 detects theparallax amount of the infrared light image 201 and the visible lightimage 202 and the motion amount between the two images.

The infrared light image imaging unit 107 and the visible light imageimaging unit 108 are two imaging units set at positions a predetermineddistance apart from each other, and thus captured images thereof (theinfrared light image 201 and the visible light image 202) are capturedfrom different viewpoints.

The same subject image is not captured in corresponding pixels, that is,pixels at the same position, of the two images from differentviewpoints, namely, the infrared light image 201 and the visible lightimage 202, and a subject shift corresponding to the parallax occurs.

Furthermore, in a case where these two images are not shot at perfectlythe same timing and the subject includes a moving subject, positions ofthe same subject captured in the respective images are different.

The parallax amount detection & motion detection unit 122 detects theparallax amount between the infrared light image 201 and the visiblelight image 202 and the motion amount between the two images, and inputsthese pieces of information, namely, parallax information and motioninformation, for example a motion vector (MV), to the image positionmatching unit 123.

(Step S304)

Processing of step S304 is executed by the image position matching unit123 illustrated in FIG. 5.

The image position matching unit 123 executes position matchingprocessing of the infrared light image 201 having been subjected to thesize adjustment and the visible light image 202 using the parallaxinformation and the motion information input from the parallax amountdetection & motion detection unit 122.

That is, two position-matched images are generated in which the samesubject is captured at the same position of the respective images.

The image position matching unit 123 outputs two images after theposition matching, that is, a post-position matching infrared lightimage 203 and a post-position matching visible light image 204illustrated in FIG. 5 to a feature amount calculating unit 124.

The post-position matching infrared light image 203 is further input toan image correcting unit 127 via a tap selection unit 125.

(Step S305)

Processing of step S305 is executed by the feature amount calculatingunit 124 illustrated in FIG. 5.

The feature amount calculating unit 124 receives the post-positionmatching infrared light image 203 and the post-position matching visiblelight image 204, extracts the feature amount of images from theseimages, and outputs the extracted feature amount data to the tapselection unit 125 and a correction parameter calculating unit 126.

As described above with reference to FIG. 6, the feature amount acquiredfrom the post-position matching infrared light image 203 and thepost-position matching visible light image 204 by the feature amountcalculating unit 124 is, for example, at least any one of the followingthree types of image feature amounts.

-   -   (1) Luminance distribution information    -   (2) Point spread function (PSF) (=function indicating a blur        mode)    -   (3) Noise information

“(1) Luminance distribution information” is distribution information ofluminance values of respective pixels in an image.

“(2) Point spread function (PSF) (=function indicating a blur mode)” isa point spread function (PSF) which is a function indicating the bluramount of an image.

“(3) Noise information” is information indicating noise included in animage.

The feature amount calculating unit 124 acquires at least one of thethree image feature amounts illustrated in FIG. 6 from the post-positionmatching infrared light image 203 and the post-position matching visiblelight image 204.

(Step S306)

Processing of step S306 is executed by the tap selection unit 125illustrated in FIG. 5.

The tap selection unit 125 executes tap setting processing to be appliedto image correction processing for improving the image quality of thepost-position matching infrared light image 203 on the basis of theimage feature amount calculated by the feature amount calculating unit124.

Specifically, the processing having been described with reference toFIGS. 5 to 14 is executed.

(Step S307)

Processing of step S307 is executed by the correction parametercalculating unit 126 illustrated in FIG. 5.

The correction parameter calculating unit 126 execute correctionparameter calculating processing to be applied to image correctionprocessing for improving the image quality of the post-position matchinginfrared light image 203 on the basis of the image feature amountcalculated by the feature amount calculating unit 124.

Specifically, the processing described above with reference to FIGS. 5to 14 is executed, and calculation of a multiplication coefficient K_(i)to be applied to a filter or other operations are performed.

(Step S308)

Processing of step S308 is executed by the image correcting unit 127illustrated in FIG. 5.

The image correcting unit 127 executes the image correction processingfor improving the image quality of the post-position matching infraredlight image 203 on the basis of the image feature amount calculated bythe feature amount calculating unit 124.

Specifically, the processing having been described with reference toFIGS. 5 to 14 is executed.

Note that, as described above, image correction processing may beexecuted by individually applying the image feature amounts of the threefeature amounts illustrated in FIG. 6, that is:

-   -   (1) Luminance distribution information;    -   (2) Point spread function (PSF) (=function indicating a blur        mode); and    -   (3) Noise information,    -   however, image correction may be performed by combining any two        or three of the above feature amounts (1) to (3).

[6. Example of Hardware Configuration of Image Processing Device]

Next, a hardware configuration example of an image processing devicewill be described with reference to FIG. 22.

FIG. 22 is a diagram illustrating a hardware configuration example of animage processing device that executes processing of the presentdisclosure.

A central processing unit (CPU) 501 functions as a control unit or adata processing unit that executes various processing according to aprogram stored in a read only memory (ROM) 502 or a storage unit 508.For example, the processing according to the sequence described in theabove embodiments is executed. In a random access memory (RAM) 503, aprogram executed by the CPU 501, data, and the like are stored. The CPU501, the ROM 502, and the RAM 503 are mutually connected by bus 504.

The CPU 501 is connected to an input/output interface 505 via the bus504. The input/output interface 505 is connected with an input unit 506,which receives a captured image of an imaging unit 521, the input unit506 including various switches, a keyboard, a mouse, a microphone, orthe like with which a user can perform input, and an output unit 507 forexecuting data output to a display unit 522, a speaker, or the like. TheCPU 501 executes various processing in accordance with an instructioninput from the input unit 506 and outputs a processing result to theoutput unit 507, for example.

The storage unit 508 connected to the input/output interface 505includes, for example, a hard disk or the like and stores a program tobe executed by the CPU 501 and various types of data. The communicationunit 509 functions as a transmission/reception unit for Wi-Ficommunication, Bluetooth (registered trademark) (BT) communication, orother data communication via a network such as the Internet and a localarea network and communicates with an external device.

A drive 510 connected to the input/output interface 505 drives aremovable medium 511 such as a magnetic disk, an optical disk, amagneto-optical disk, a semiconductor memory such as a memory card andexecutes recording or reading of data.

[7. Summary of Configurations of the Present Disclosure]

As described above, the embodiments of the present disclosure have beendescribed in detail with reference to specific embodiments. However, itis obvious that those skilled in the art can make modifications orsubstitutions of the embodiments within a scope not departing from theprincipals of the present disclosure. That is, the present invention hasbeen disclosed in the form of exemplification and thus should not beinterpreted in a limited manner. In order to judge the gist of thepresent disclosure, Claims should be taken into consideration.

Note that the technology disclosed herein may also employ configurationsas follows.

(1) An image processing device, including:

-   -   a feature amount calculating unit for receiving an infrared        light image and a visible light image and extracting a feature        amount from at least one of the images; and    -   an image correcting unit for executing pixel value correction        processing on the infrared light image on the basis of a        reference area and a correction parameter determined depending        on the feature amount.

(2) The image processing device according to item (1), furtherincluding:

-   -   a tap selection unit for determining the reference area used for        the pixel value correction in the image correcting unit on the        basis of the feature amount; and    -   a correction parameter calculating unit for determining the        correction parameter used for the pixel value correction in the        image correcting unit on the basis of the feature amount,    -   in which the image correcting unit executes the pixel value        correction processing in which a tap determined by the tap        selection unit and the correction parameter determined by the        correction parameter calculating unit are applied.

(3) The image processing device according to item (1) or (2),

-   -   in which the feature amount calculating unit extracts any one of        the following feature amounts (a) to (c):    -   (a) luminance distribution information;    -   (b) blur mode information; and    -   (c) noise information,    -   from at least one of the infrared light image and the visible        light image.    -   (4) The image processing device according to any one of        items (1) to (3),    -   in which the feature amount calculating unit acquires luminance        distribution information from the infrared light image and the        visible light image, and    -   the image correcting unit executes the pixel value correction        processing of the infrared light image such that a shift between        a luminance distribution of the infrared light image and that of        the visible light image is eliminated.

(5) The image processing device according to item (4), furtherincluding:

-   -   a tap selection unit for determining the reference area used for        the pixel value correction in the image correcting unit on the        basis of the feature amount; and    -   a correction parameter calculating unit for determining the        correction parameter used for the pixel value correction in the        image correcting unit on the basis of the feature amount,    -   in which the tap selection unit sets a wider reference pixel        area (tap area) as the shift between the luminance distribution        of the infrared light image and that of the visible light image        is large,    -   the correction parameter calculating unit determines a        correction parameter for eliminating the shift between the        luminance distribution of the infrared light image and that of        the visible light image, and    -   the image correcting unit executes the pixel value correction        processing in which a tap determined by the tap selection unit        and the correction parameter determined by the correction        parameter calculating unit are applied.

(6) The image processing device according to any one of items (1) to(5),

-   -   in which the feature amount calculating unit acquires blur mode        information from the infrared light image, and    -   the image correcting unit executes the pixel value correction        processing of the infrared light image such that a blur in the        infrared light image is reduced.

(7) The image processing device according to item (6), furtherincluding:

-   -   a tap selection unit for determining the reference area used for        the pixel value correction in the image correcting unit on the        basis of the feature amount; and    -   a correction parameter calculating unit for determining the        correction parameter used for the pixel value correction in the        image correcting unit on the basis of the feature amount,    -   in which the tap selection unit sets a wider reference pixel        area (tap area) as a blurred range of the infrared light image        is large,    -   the correction parameter calculating unit determines a        correction parameter for eliminating the blur in the infrared        light image, and    -   the image correcting unit executes the pixel value correction        processing in which a tap determined by the tap selection unit        and the correction parameter determined by the correction        parameter calculating unit are applied.

(8) The image processing device according to any one of items (1) to(7),

-   -   in which the feature amount calculating unit acquires noise        information from the visible light image, and    -   the image correcting unit executes the pixel value correction        processing of the infrared light image such that noise in the        infrared light image is reduced.

(9) The image processing device according to item (8), furtherincluding:

-   -   a tap selection unit for determining the reference area used for        the pixel value correction in the image correcting unit on the        basis of the feature amount; and    -   a correction parameter calculating unit for determining the        correction parameter used for the pixel value correction in the        image correcting unit on the basis of the feature amount,    -   in which the tap selection unit sets a wider reference pixel        area (tap area) as noise in the visible light image includes        more noise having a low frequency band component,    -   the correction parameter calculating unit determines a        correction parameter corresponding to a noise component included        in the visible light image, and    -   the image correcting unit executes the pixel value correction        processing in which a tap determined by the tap selection unit        and the correction parameter determined by the correction        parameter calculating unit are applied.

(10) The image processing device according to any one of items (1) to(9),

-   -   in which the image correcting unit executes the image correction        processing on the visible light image by applying a corrected        infrared light image acquired as a result of the pixel value        correction of the infrared light image.

(11) An imaging device, including:

-   -   an infrared light image imaging unit for performing imaging        processing of an infrared light image;    -   a visible light image imaging unit for performing imaging        processing of a visible light image; and    -   an image processing unit for receiving the infrared light image        and the visible light image and executing pixel value correction        processing of at least one of the images,    -   in which the image processing unit includes: a feature amount        calculating unit for receiving the infrared light image and the        visible light image and extracting a feature amount from at        least one of the images; and    -   an image correcting unit for executing pixel value correction        processing on the infrared light image on the basis of a        reference area and a correction parameter determined depending        on the feature amount.

(12) The imaging device according to item (11),

-   -   in which the image processing unit further includes:    -   a tap selection unit for determining the reference area used for        the pixel value correction in the image correcting unit on the        basis of the feature amount; and    -   a correction parameter calculating unit for determining the        correction parameter used for the pixel value correction in the        image correcting unit on the basis of the feature amount, and    -   the image correcting unit executes the pixel value correction        processing in which a tap determined by the tap selection unit        and the correction parameter determined by the correction        parameter calculating unit are applied.

(13) The imaging device according to item (11) or (12),

-   -   in which the feature amount calculating unit extracts any one of        the following feature amounts (a) to (3 c:    -   (a) luminance distribution information;    -   (b) blur mode information; and    -   (c) noise information,    -   from at least one of the infrared light image and the visible        light image.

(14) An image processing device, including:

-   -   a feature amount calculating unit for receiving an infrared        light image and extracting a feature amount; and    -   an image correcting unit for executing pixel value correction        processing on the infrared light image on the basis of a        reference area and a correction parameter determined depending        on the feature amount.

(15) The image processing device according to item (14), furtherincluding:

-   -   a tap selection unit for determining the reference area used for        the pixel value correction in the image correcting unit on the        basis of the feature amount; and    -   a correction parameter calculating unit for determining the        correction parameter used for the pixel value correction in the        image correcting unit on the basis of the feature amount,    -   in which the image correcting unit executes the pixel value        correction processing in which a tap determined by the tap        selection unit and the correction parameter determined by the        correction parameter calculating unit are applied.

(16) The image processing device according to item (14) or (15),

-   -   in which the feature amount calculating unit extracts any one of        the following feature amounts (a) to (c):    -   (a) luminance distribution information;    -   (b) blur mode information; and    -   (c) noise information,    -   from the infrared light image.

(17) An image processing method executed in an image processing device,the method including:

-   -   a feature amount calculating step of receiving, by a feature        amount calculating unit, an infrared light image and a visible        light image and extracting a feature amount from at least one of        the images; and    -   an image correcting step of executing, by an image correcting        unit, pixel value correction processing on the infrared light        image on the basis of a reference area and a correction        parameter determined depending on the feature amount.

(18) A program for causing an image processing device to execute imageprocessing,

-   -   in which the program causes a feature amount calculating unit to        receive an infrared light image and a visible light image and to        extract a feature amount from at least one of the images; and    -   the program causes an image correcting unit to execute pixel        value correction processing on the infrared light image on the        basis of a reference area and a correction parameter determined        depending on the feature amount.

Meanwhile, a series of processing described herein can be executed byhardware, software, or a composite configuration thereof. In a casewhere processing by software is executed, a program storing a processingsequence may be installed in a memory in a computer incorporated indedicated hardware and thereby executed. Alternatively, the program maybe installed in a general-purpose computer capable of executing varioustypes of processing and be executed. For example, the program can berecorded in a recording medium in advance. In addition to installationfrom a recording medium to a computer, the program can be received via anetwork such as a local area network (LAN) and the Internet andinstalled on a recording medium such as a built-in hard disk.

Note that the various processing described herein may be executed notonly in time series in accordance with the description but also inparallel or separately depending on the processing capability of adevice executing the processing or as necessary. In addition, in thisspecification, the term “system” refers to a logical group configurationof a plurality of devices, and is not limited to those in which devicesof respective components are in the same housing.

INDUSTRIAL APPLICABILITY

As described above, according to a configuration of an embodiment of thepresent disclosure, a device and a method for executing image qualityimprovement processing of an infrared light image are implemented.

Specifically, included are: a feature amount calculating unit forreceiving an infrared light image and a visible light image andextracting a feature amount from at least one of the images; and animage correcting unit for executing pixel value correction processing onthe infrared light image on the basis of a reference area and acorrection parameter determined depending on the feature amount. Furtherincluded are: a tap selection unit for determining the reference areaused for the pixel value correction on the basis of the feature amount;and a correction parameter calculating unit for determining thecorrection parameter used for the pixel value correction on the basis ofthe feature amount. The image correcting unit executes the pixel valuecorrection processing in which a tap determined by the tap selectionunit and the correction parameter determined by the correction parametercalculating unit are applied.

By these flows of processing, a device and a method for executing imagequality improvement processing of an infrared light image areimplemented.

REFERENCE SIGNS LIST

-   10 Visible light image-   20 Infrared light image-   100 Image processing device-   101 Control unit-   102 Storage unit-   103 Codec-   104 Input unit-   105 Output unit-   106 Imaging unit-   107 Infrared light image imaging unit-   108 Visible light image imaging unit-   111 First imaging element-   112 Second imaging element-   113 Infrared light (IR) emitting unit-   120 Image processing unit-   121 Scaler-   122 Parallax amount detection & motion amount detection unit-   123 Image position matching unit-   124 Feature amount calculating unit-   125 Tap selection unit-   126 Correction parameter calculating unit-   127 Image correcting unit-   131 Infrared light image correcting unit-   132 Visible light image correcting unit-   151 Image reducing unit-   161 Pseudo-infrared light image generating unit-   171 Subtractor-   172 Adder-   181 Feature amount calculating unit-   182 Tap selection unit-   183 Correction parameter calculating unit-   184 Image correcting unit-   191 Corrected infrared light image-   192 Visible light image-   501 CPU-   502 ROM-   503 RAM-   504 Bus-   505 Input/output interface-   506 Input unit-   507 Output unit-   508 Storage unit-   509 Communication unit-   510 Drive-   511 Removable medium-   521 Imaging unit-   522 Display unit

1. An image processing device, comprising: a feature amount calculatingunit for receiving an infrared light image and a visible light image andextracting a feature amount from at least one of the images; and animage correcting unit for executing pixel value correction processing onthe infrared light image on the basis of a reference area and acorrection parameter determined depending on the feature amount.
 2. Theimage processing device according to claim 1, further comprising: a tapselection unit for determining the reference area used for the pixelvalue correction in the image correcting unit on the basis of thefeature amount; and a correction parameter calculating unit fordetermining the correction parameter used for the pixel value correctionin the image correcting unit on the basis of the feature amount, whereinthe image correcting unit executes the pixel value correction processingin which a tap determined by the tap selection unit and the correctionparameter determined by the correction parameter calculating unit areapplied.
 3. The image processing device according to claim 1, whereinthe feature amount calculating unit extracts any one of the followingfeature amounts (a) to (c): (a) luminance distribution information; (b)blur mode information; and (c) noise information, from at least one ofthe infrared light image and the visible light image.
 4. The imageprocessing device according to claim 1, wherein the feature amountcalculating unit acquires luminance distribution information from theinfrared light image and the visible light image, and the imagecorrecting unit executes the pixel value correction processing of theinfrared light image such that a shift between a luminance distributionof the infrared light image and that of the visible light image iseliminated.
 5. The image processing device according to claim 4, furthercomprising: a tap selection unit for determining the reference area usedfor the pixel value correction in the image correcting unit on the basisof the feature amount; and a correction parameter calculating unit fordetermining the correction parameter used for the pixel value correctionin the image correcting unit on the basis of the feature amount, whereinthe tap selection unit sets a wider reference pixel area (tap area) asthe shift between the luminance distribution of the infrared light imageand that of the visible light image is large, the correction parametercalculating unit determines a correction parameter for eliminating theshift between the luminance distribution of the infrared light image andthat of the visible light image, and the image correcting unit executesthe pixel value correction processing in which a tap determined by thetap selection unit and the correction parameter determined by thecorrection parameter calculating unit are applied.
 6. The imageprocessing device according to claim 1, wherein the feature amountcalculating unit acquires blur mode information from the infrared lightimage, and the image correcting unit executes the pixel value correctionprocessing of the infrared light image such that a blur in the infraredlight image is reduced.
 7. The image processing device according toclaim 6, further comprising: a tap selection unit for determining thereference area used for the pixel value correction in the imagecorrecting unit on the basis of the feature amount; and a correctionparameter calculating unit for determining the correction parameter usedfor the pixel value correction in the image correcting unit on the basisof the feature amount, wherein the tap selection unit sets a widerreference pixel area (tap area) as a blurred range of the infrared lightimage is large, the correction parameter calculating unit determines acorrection parameter for eliminating the blur in the infrared lightimage, and the image correcting unit executes the pixel value correctionprocessing in which a tap determined by the tap selection unit and thecorrection parameter determined by the correction parameter calculatingunit are applied.
 8. The image processing device according to claim 1,wherein the feature amount calculating unit acquires noise informationfrom the visible light image, and the image correcting unit executes thepixel value correction processing of the infrared light image such thatnoise in the infrared light image is reduced.
 9. The image processingdevice according to claim 8, further comprising: a tap selection unitfor determining the reference area used for the pixel value correctionin the image correcting unit on the basis of the feature amount; and acorrection parameter calculating unit for determining the correctionparameter used for the pixel value correction in the image correctingunit on the basis of the feature amount, wherein the tap selection unitsets a wider reference pixel area (tap area) as noise in the visiblelight image includes more noise having a low frequency band component,the correction parameter calculating unit determines a correctionparameter corresponding to a noise component included in the visiblelight image, and the image correcting unit executes the pixel valuecorrection processing in which a tap determined by the tap selectionunit and the correction parameter determined by the correction parametercalculating unit are applied.
 10. The image processing device accordingto claim 1, wherein the image correcting unit executes the imagecorrection processing on the visible light image by applying a correctedinfrared light image acquired as a result of the pixel value correctionof the infrared light image.
 11. An imaging device, comprising: aninfrared light image imaging unit for performing imaging processing ofan infrared light image; a visible light image imaging unit forperforming imaging processing of a visible light image; and an imageprocessing unit for receiving the infrared light image and the visiblelight image and executing pixel value correction processing of at leastone of the images, wherein the image processing unit comprises: afeature amount calculating unit for receiving the infrared light imageand the visible light image and extracting a feature amount from atleast one of the images; and an image correcting unit for executingpixel value correction processing on the infrared light image on thebasis of a reference area and a correction parameter determineddepending on the feature amount.
 12. The imaging device according toclaim 11, wherein the image processing unit further comprises: a tapselection unit for determining the reference area used for the pixelvalue correction in the image correcting unit on the basis of thefeature amount; and a correction parameter calculating unit fordetermining the correction parameter used for the pixel value correctionin the image correcting unit on the basis of the feature amount, and theimage correcting unit executes the pixel value correction processing inwhich a tap determined by the tap selection unit and the correctionparameter determined by the correction parameter calculating unit areapplied.
 13. The imaging device according to claim 11, wherein thefeature amount calculating unit extracts any one of the followingfeature amounts (a) to (c): (a) luminance distribution information; (b)blur mode information; and (c) noise information, from at least one ofthe infrared light image and the visible light image.
 14. An imageprocessing device, comprising: a feature amount calculating unit forreceiving an infrared light image and extracting a feature amount; andan image correcting unit for executing pixel value correction processingon the infrared light image on the basis of a reference area and acorrection parameter determined depending on the feature amount.
 15. Theimage processing device according to claim 14, further comprising: a tapselection unit for determining the reference area used for the pixelvalue correction in the image correcting unit on the basis of thefeature amount; and a correction parameter calculating unit fordetermining the correction parameter used for the pixel value correctionin the image correcting unit on the basis of the feature amount, whereinthe image correcting unit executes the pixel value correction processingin which a tap determined by the tap selection unit and the correctionparameter determined by the correction parameter calculating unit areapplied.
 16. The image processing device according to claim 14, whereinthe feature amount calculating unit extracts any one of the followingfeature amounts (a) to (c): (a) luminance distribution information; (b)blur mode information; and (c) noise information, from the infraredlight image.
 17. An image processing method executed in an imageprocessing device, the method comprising: a feature amount calculatingstep of receiving, by a feature amount calculating unit, an infraredlight image and a visible light image and extracting a feature amountfrom at least one of the images; and an image correcting step ofexecuting, by an image correcting unit, pixel value correctionprocessing on the infrared light image on the basis of a reference areaand a correction parameter determined depending on the feature amount.18. A program for causing an image processing device to execute imageprocessing, wherein the program causes a feature amount calculating unitto receive an infrared light image and a visible light image and toextract a feature amount from at least one of the images, and theprogram causes an image correcting unit to execute pixel valuecorrection processing on the infrared light image on the basis of areference area and a correction parameter determined depending on thefeature amount.